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(54) System and method for obtaining prices for items 



(57) The invention relates to a system and method 
for obtaining prices for items which are too small to carry 
readable bar code labels. The system includes a cam- 
era (18) which records an image of an item (22), and 



data processing means (15,16) which is coupled to the 
camera (18) and which identifies the item (22) from the 
image and which obtains the price of the item (22) from 
a price-lookup file (36). 
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Description 



The present invention relates to object identification systems, and more specifically to a system and method for 
obtaining prices for items. 

Readable bar code labels are difficult or impossible to attach to fasteners and other small unpacked Items For 
example, in a typical building supply store, a store clerk must identify small items by visually matching a customer- 
provided item to one of a plurality of sample items fastened to a sheet of cardboard, or by manually identifying the item 
in a blue-print book. The clerk reads an item number, such as a SKU number, for the identified item from the cardboard 
sheet or blue-print book, and enters the item number into the transaction using a keyboard of a retail terminal Alter- 
natively, the clerk may scan the bar code next to a picture of the item in a book. These methods are time consuming 
and subject to error. 3 

Most retailers realize that unpacked items increase checkout time. They tend to package most of the small items 
in boxes, forcing the customers to purchase the items in a quantity that sometimes is unnecessary and even wasteful 

It is an object of the invention to provide a system and method for improving check-out speeds for transactions 
involving items that are too small to carry readable bar code labels, and without the need for purchasing such items 
in unnecessary quantity. 

According to one aspect of the invention there is provided a system for obtaining prices for items, characterized 
by a camera which records an image of an item, and data processing means which is coupled to said camera and 
which identifies the item from said image and which obtains the price of the item from a price-lookup file 

According to another aspect of the invention there is provided a method of obtaining a price of an item characterized 
by the steps of capturing an image of the item by a camera, extracting predetermined features from the image by data 
processing means, executing a parsing algorithm to identify the item from corresponding features in a feature database 
by said data processing means, determining an identification number for the item from the feature database by said 
data processing means, and obtaining a description of the item and its price from a price-lookup file by said data 
processing means. 

In one embodiment of the invention, a system includes an image processing computer coupled to the camera 
which identifies the item from the image, a transaction server coupled to the image processing server which obtains 
the price from a price-lookup file, and a transaction terminal coupled to the transaction server and located in proximity 
with the camera which completes a transaction using the price information. 

The system may further include a plurality of additional transaction terminals coupled to the transaction server and 
a plurality of additional cameras located in proximity with the additional transaction terminals for producing a plurality 
of additional images. In such a system, each camera preferably includes an operator switch for signaling the image 
processing server to activate the camera and for identifying the transaction terminal associated with the camera The 
image processing server controls processing of images from individual cameras through a multiplexor 

The method of obtaining a price ot an item is based upon an analysis of features extracted from a captured image 
of the item. A parsing algorithm identifies the item from corresponding features in a feature database. The image 
processing server determines an identification number for the item from the feature database. The transaction server 
obtains the price from a PLU file and forwards it to the terminal associated with a requesting camera 

An embodiment of the present invention will now be described by way of example with reference to the accompa- 
nying drawings, in which: ^ 

Figs. 1 A and 1 B form a block diagram of the item recognition system of the present invention; 
Fig. 2 is a perspective view of a camera assembly; 

Figs. 3A and 3B form an example of a parsing diagram for single-boundary items used by the recognition system- 
Figs. 4A and 4B form an example of a parsing diagram for two-boundary items used by the recognition system* ' 
Fig. 5 is a flow diagram illustrating the operation of the system in Fig. 1 ; and 

Fig. 6 is a block diagram of an alternative embodiment of the item recognition system of the present invention. 

Referring now to Fig. 1, system 10 primarily includes camera assembly 12, terminal 14, image processing server 
15, and transaction server 16. System 10 may also include additional peripherals, including bar code reader 66 

Camera assembly 12 includes camera IB and light 19. Camera 1 B is preferably a commercially available charge- 
coupled device (CCD) camera, such as one produced by Sensormatic, Inc., which records pixel images 20 of item 22 
and which signals image processing server 15 with information identifying the terminal associated with camera 18 
Camera 18 includes a focal plane array consisting of a two-dimensional array of pixels. Camera 18 is preferably used 
in combination with bar code reader 66, due to the processing limitations of terminal 1 4, but on more powerful systems 
it may be used without bar code reader 66 to capture images of items with and without bar code labels 

Camera assembly 12 further includes switch 26. When engaged, switch 26 sends a TERMINAL ID for the asso- 
ciated 90S terminal 14 and a recognition request to image processing server 15. 
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Server 15 returns a "start" signal to activate camera 18. 
Light 19 illuminates item 22. 

Preferably, a plurality of camera assemblies 12 is located throughout the transaction establishment. Video data 
cables and a control cable from each camera assembly 12 are multiplexed by multiplexor 38 into a frame grabber 
adapter card 39 within image processing server 15. Frame grabber adapter card 39 digitizes the images 20 from 
cameras 18. 

Terminal 14 includes processor 24, display 28, input device 30, and printer 32, although known additions, deletions, 
and substitutions to this configuration are also envisioned within the meaning of the word terminar. 

Processor 24 executes transaction processing software 34 to support transaction processing. For example, trans- 
action processing software 34 obtains the prices of all merchandise items, including prices of item 22 identified by 
camera 18, from a price look-up (PLU) file 36 associated with transaction server 16. Transactbn processing software 
34 tallies the prices of the items and directs printer 32 to print a receipt to complete the transaction. 

Input device 30 is preferably a keyboard. 

Bar code reader 66 reads bar code labels on items having bar code labels. Preferably, bar code reader 66 is an 
optical bar code reader. Bar code reader 66 returns a SKU number 64 to processor 24. 

Image processing server 15 processes images 20. Processor 68 executes frame grabber software 40 and image 
processing software 42. Frame grabber software 40 is a driver that controls camera 18, produces gray-scale image 
44 from pixel image 20, and stores gray-scale image 44 in memory 26. 

Image processing software 42 includes image thresholder 46, feature extractor 48, and item identifier 50. 

Image thresholder 46 converts gray-scale image 44 from frame grabber software 40 to binary image 52 using well- 
known algorithms. If the pixel gray level is greater than the threshold value, the pixel is assigned a pixel value of M u , 
otherwise it is assigned a pixel value of n O\ Binary image 52 is a compacted version of the original pixel images 20, 
since every eight original gray-scale pixels (eight bytes) are now packed in one byte with one bit representing one pixel. 

Featu re extractor 48 extracts features 54 from binary image 52. 1 n this context, f ealu res 54 are defined as something 
that can be numerically computed from binary image 52, either directly or indirectly. 

Features 54 include both direct and indirect features. Features 54 are direct features if they can be extracted 
directly from binary image 52. For example, the shaft length and shaft radius of a nail are considered direct features. 
Usually, the indirect features pertain to some mathematical properties that make different items easier to distinguish 
than by using the direct features alone. For example, where both a cement nail and a flat head nail may have a similar 
head width or head radius, the two nails can be distinguished by comparing the ratio of head width to head radius for 
the two nails. This ratio is used as an indirect feature and shows that the ratio from the cement nail is larger than the 
ratio from the common flat head nail. 

A small item usually possesses several leatures that can be used later on in the identification process. For example, 
the nail has a boundary (contour shape), shaft length, shaft radius, head width, and head radius. A washer has different 
features, namely first and second boundaries, outer and inner boundary radii, co-centered first and second boundaries, 
and circular first and second boundaries. 

Feature extractor 48 provides an array of features 54 that represent item 22. At this point, binary image 52 no 
longer contains any useful information and can be discarded from memory 26 if memory 26 is limited in size. Since 
storing an image usually requires a large memory space, it is not practical to continuously operate on binary image 52. 

Feature extractor 48 provides useful information regarding binary image 52 in a more compact format. In addition 
to using less of memory 26, features 54 are easier to work with. 

Item identifier 50 executes a parsing algorithm that compares features 54 to features stored in feature database 
33 to identify item 22 and produce a SKU number output 58. Item identifier 50 sends the SKU number and the identity 
of the terminal associated with the camera producing image 20 to transaction server 16. 

Memory 26 stores software, gray-scale image 44, binary image 52, features 54, output 58, and reference features 

56." 

Storage medium 70 stores feature database 33 and is preferably a fixed disk drive. Feature database 33 contains 
reference features 56 on items 22 within a transaction establishment. 

Transaction server 16 processes requests for price information from terminal 14 and image processing server 15. 
Transaction server 16 receives SKU numbers from image processing server 15 and from terminal 14. Transaction 
server 16 reads PLU file 36 and transmits corresponding price information to terminal 14. Image processing server 15 
sends information identifying the terminal associated with the camera in use so that transactbn server 16 may route 
the SKU numbers to that terminal. 

Transaction server 16 includes storage medium 72, which stores PLU file 36. Storage medium 72 is preferably a 
fixed disk drive. 

Terminal 14, image processing server 15, and transaction server 16 are preferably part of a network and linked in 
a known manner. Of course, image processing server 15 and transaction server 16 may be the same computer. 
With reference to Fig. 6, image processing server 15 may be eliminated and the functions ot image processing 
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server 15 may be executed instead by terminal 14. For example, frame grabber card 39 may include a digital signal 
processor or other processing circuitry to manage image processing chores within terminal 14. Operation of camera 
18 may be started by a user by striking a key on terminal 14 or by engaging a button on camera 18. This example 
would avoid the need to multiplex image camera connections and the need to send a terminal address with an image 
5 processing request. 

In addition, any ol the above computers may use image compression as necessary to speed transfer and process- 
ing of images. For example, an item image may be captured by camera 1 8, digitized and compressed by a digital signal 
processor or state machine, and then sent to terminal 14 for analysis. 

Finally, other methods of identifying items may be used in conjunction with the system of the present invention. 

io Thus, the system may additionally include a small scale and/or an electromagnet. The scale does not have to be very 
precise, since it is intended to be used to compare the weight when the electromagnet is on and off to determine 
whether the object is magnetic or not. "Hi is enables the device to recognize the difference between steel and aluminum 
screws. A switchable filter might be necessary to do a primitive color filtering comparison to resolve the difference 
between aluminum and brass since both are not magnetic. 

is Once it identifies item 22, item identifier 50 sends the SKU number to transaction processing software 34. 

An alternative processing method involves the use of a chain code to represent a boundary of item 22. A chain 
code is a connected sequence of straight line segments. Their use in digital image processing is well-known in the art. 
See for example, "Digital Image Processing', by Rafael C. Gonzalez and Paul Wintz, Chapter 8.1.1, pages 392-395. 
This reference is hereby incorporated by reference. Once terminal 14 has determined a chain code representing the 

20 boundary of item 22, terminal 1 4 may then compare the chain code to previously stored chain codes in a chain code 
database. 

Turning now to Fig. 2, camera assembly 1 2 is shown in more detail. Camera assembly 1 2 couples to image process- 
ing server 15 through cable 86. Cable 86 includes individual image and control lines. 

Camera assembly 12 includes base portion 80 and lid portion 82. Base portion 80 contains cavity 84. 
25 Lid 82 contains camera 1 8 and is hinged to base portion 80. 

if camera 18isaCCDcamera, then light 19 is mounted at the bottom of the box, just underthe parttobe recognized. 
Of course, there may be other configurations based upon the type of camera system. 

Camera assembly 12 includes button 87 which controls switch 26. 

With reference to Figs. 3A and 3B, a parsing diagram for one boundary item is shown beginning with step 88. 
30 Using this parsing diagram, item identifier 50 is able to identify parts including an alien head cap screw 94, hex bolt 
96, flat head screw 104, round head screw 106, flat head nail 110, cement nail 112, flat head machine screw 122, 
round head machine screw 126, carriage bolt 12B, alien screw 116, and finishing nail 118. Of course, this parsing 
diagram is illustrative of the process. Other items may also be identified with similar parsing diagrams. 

Parts 104, 106, and 122 may be identified using only direct features. However, parts 94, 96, 110, 112, 116, 118, 
35 126, and 1 28 may be identified if indirect features are examined. 

Direct features are represented in steps 90, 98, 100, 102, and 120. In step 90, the parsing algorithm determines 
whether a part has a head and the type of head: hex or alien, or round or flat. Step 98 determines whether a round or 
flat-headed part has a tip. Step 100 determines whether a round or flat-headed part with a tip has a thread. Step 102 
determines whether the round or flat-headed part with a tip and a thread has a flat head. Finally, step 120 determines 
40 whether a round or flat-headed part without a tip has a flat head. 

Indirect features are represented in steps 92, 108, 114, and 124. In step 92, the parsing algorithm determines 
whether a part with a hex or alien head has a head radius to shaft radius ratio less than a predetermined threshold. If 
it does, the part is an alien head cap screw 94. If it does not, the part is a hex bolt 96. 

In step 1 08, the parsing algorithm determines whether a part with a round or flat head and a tip but no thread has 
45 a shaft radius to shaft length ratio less than a predetermined threshold. If it does, the part is a flat head nail 110. If it 
does not, the part is a cement nail 112. 

In step 114, the parsing algorithm determines whether a part without a head has a shaft radius to shaft length ratio 
less than a predetermined threshold. If it does, algorithm 100 checks whether the part has threads; if it has, the part 
is an alien screw 116; otherwise, it is a pin 115. On the other hand, if the shaft radius to shaft length ratio is not less 
so than the threshold, the part is a finishing nail 118. 

Finally, in step 124, the parsing algorithm determines whether a part with a round head and no tip has a head 
radius to shaft length ratio less than a predetermined threshold. If it does, the part is a round head machine screw 1 26. 
If it does not, the part is a carriage bolt 1 28. 

With reference to Figs. 4A and 4B, a parsing diagram for two-boundary items is shown beginning with START 1 30. 
55 Using this parsing diagram, item identifier 50 is able to identify parts including a flat washer 138, a lock washer 142, 
a wing nut 144, a square nut 146, a hex nut 148, an octagon nut 150, an external star washer 152, an internal star 
washer 1 56, a cast eye bolt 1 62, a turned eye bolt 164, and a cotter pin 1 66. Of course, this parsing diagram is illustrative 
of the process. Other items may also be identified with similar parsing diagrams. 
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Parts 138, 156, 162, 164, and 166 may be identified using only direct features. However, parts 142-152 may be 
identified if indirect features are examined as well. 

Direct features are represented in steps 1 32, 1 34, 1 36, 1 54, and 1 60. In step 1 32, the parsing algorithm determines 
whether the two boundaries are co-centered. Steps 1 34 and 1 60 determine whether the inner boundary is a circle. 
5 Steps 1 36 and 1 54 determine whether the outer boundary is a ci rcle. 

Thus, if item 22 has two co-centered boundaries and the inner and outer boundaries are both circles, then the 
parsing algorithm identifies item 22 as a flat washer 133. 

If item 22 has two co-centered boundaries, but only the outer boundary is a circle, then the parsing algorithm 
identifies item 22 as an internal star washer 156. 
10 If item 22 does not have two co-centered boundaries, but the inner boundary is a circle, then the parsing algorithm 
identifies item 22 as a cat eye boft 162. 

If item 22 does not have two co-centered boundaries, and the inner boundary is not a circle, then the parsing 
algorithm identifies item 22 as a cotter pin 166. 

Indirect features are represented in steps 140 and 160. In step 140, the parsing algorithm determines the number 
'5 of extremes of the outer boundary from the center of the item. In step 160, the parsing algorithm determines the close- 
ness of the inner boundary to a circle. 

Thus, if item 22 does not have two co-centered boundaries, and the inner boundary is almost a circle, then the 
parsing algorithm identifies item 22 as a turned eye bolt 1 64. 

If item 22 has two co-centered boundaries and only the inner boundary is a circle, then the parsing algorithm 
20 examines the extreme count to identify item 22. If the extreme count is less the two, the parsing algorithm identifies 
item 22 as lock washer 142. If the extreme count is two, the parsing algorithm identifies item 22 as wing nut 144. If the 
extreme count is four, the parsing algorithm identifies item 22 as square nut 1 46. If the extreme count is six, the parsing 
algorithm identifies item 22 as hex nut 148. II the extreme count is eight, the parsing algorithm identifies item 22 as 
octagon nut 150. If the extreme count is greater than eight, the parsing algorithm identifies item 22 as an external star 
2S washer 152, 

With reference to Fig. 5, the operation of system 10 is described in detail beginning with START 170. 
In step 172, a clerk places item 22 within cavity 84 and closes lid portion 82. 

In step 1 74, camera assembly 1 2 sends a terminal ID and request for item recognition to image processing server 
15 upon engagement of switch 26 by the clerk. 
30 in step 178, if image processing server 1 5 is available, it switches multiplexor 38 to connect frame grabber adapter 

card 39 to the camera 18 associated with the POS terminal 1 4 having the sent terminal ID and activates camera 18. 

In step 180, frame grabber software 40 captures pixel image 20 and produces gray-scale image 44. 

In step 182, image thresholder 46 converts gray-scale image 44 to binary image 52. 

In step 184, feature extractor 48 extracts predetermined features 54 from binary image 52. 
35 In step 1 86, item identifier 50 determines whether item 22 has one or two boundaries from features 54. 

In step 188, item identifier 50 executes the parsing algorithm of Figs. 3A and 3B for a single-boundary item or the 
parsing algorithm of Figs. 4A and 4B for a two-boundary item to identify item 22 from features 54. 

During this step, item identifier 50 preferably converts features 54 to descriptions that are more familiar to ordinary 
people. This is because the direct features are measured in pixels, while the items in a hardware store are normally 
40 measured in inches or centimeters and rounded to some specific values, such as 1/16", t/8\ 1/4', 1/2", etc. 

The direct features may also vary by a predetermined amount about a standard value. Therefore, item identifier 
50 preferably creates a look-up table to convert part sizes from pixels to inches and quantize sizes to standard sizes. 
For instance, the following look-up table converts feature information for a cement nail 112: 



50 



Shaft Length Range 


Standard Shaft Length 


SKU Number 


3.2 -3.3 in. 


3.25 in. 


111111 


4.25 -4.75 in. 


4.5 in. 


222222 


5.5 - 6.5 in. 


6 in. 


333333 i 


Look-up Table 



In step 1 90, item identifier 50 determines a SKU number for item 22 from feature database 33. For items having various 
sizes or dimensions, item identifier 50 compares the determined dimension of item 22 to values in a lookup table. In 
the example above, item identifier 50 compares the length of cement nail 112 determined from binary image 52 to each 
of the three standard shaft lengths in the table to determine which of the three SKU numbers to report to transaction 
server 16. 

In step 192, item identifier 50 sends a message addressed to the terminal 14 associated with the TERMINAL ID 
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and containing the SKU number to transaction server t6. 

In step 1 94, transaction server 1 6 obtains a description and price for item 22 from PLU file 36. 
In step 1 96, transaction server 1 6 forwards the description and the price for item 22 to terminal 1 4. 
In step 19B, terminal 14 adds the description and price to the transaction. - 
5 In step 200, the method ends 



Claims 

io 1 A system for obtaining prices for Items (22), characterized by a camera (1 8) which records an image of an item 
(22), and data processing means (15,16) which is coupled to said camera and which identifies the item from said 
image and which obtains the price of the item from a price-lookup file (36). 

2. A system according to claim 1 , characterized in that said camera (1 8) is a charge coupled device camera. 

75 

3. A system according to either claim 1 or claim 2, characterized by a container including a base portion (80) and a 
lid portion (B2), wherein said camera (18) is located within said container, and wherein said base portion has a 
cavity (84) in which the item (22) is placed. 

20 4. A system according to claim 3, characterized in that said lid portion (82) contains said camera (18). 

5. A system according to any one of the preceding claims, characterized by a feature database (33) which is coupled 
tosaid data processing means (15,16) and which contains features describing the item (22). 

2S 6. A system according to any one of the preceding claims, characterized in that said data processing means (1 5, 1 6) 
includes a frame grabber adapter (39) arranged to digitize the image o1 the item (22) recorded by said camera (18). 

7. A system according to claim 6, characterized by a plurality of additional cameras coupled to said frame grabber 
adapter (39) which is arranged to digitize the image of an item recorded by any of said additional cameras. 

30 

8. A system according to any one of the preceding claims, characterized by at least one transaction terminal (14) 
coupled to said data processing means (1 5, 1 6). 

9. A method of obtaining a price of an item (22), characterized by the steps of capturing an image of the item (22) 
35 by a camera (18), extracting predetermined features from the image by data processing means (1 5,16), executing 

a parsing algorithm to identify the item from corresponding features in a feature database (33) by said data process- 
ing means (15,16), determining an identification number for the item from the feature database by said data 
processing means (15,16), and obtaining a description of the item and its price from a price-lookup file (36) by 
said data processing means. 

40 

10. A method according to claim 9, characterized by the steps of sending a message identifying a transaction terminal 
(1 4) and including a request for item recognition to said data processing means (15,16), connecting a frame grabber 
adapter (39) included in said data processing means to said camera (1 B), signaling said camera to record an image 
of the item by said data processing means, forwarding said description and price to said transaction terminal (14) 

45 by said data processing means, and adding said description and price to a transaction by said transaction terminal. 



50 



S5 



6 



EP0 843 293 A2 




7 



EP 0 B43 293 A2 




EP 0 843 293 A2 



FIG. 2 
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FIG. 3A 
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FIG. 3B 
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FIG. 4B 
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FIG. 5 



ITEM 12 IS PLACED WITHIN CAVITY84] ^ 172 
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CAMERA ASSEMBLY 12 SENDS A TERMINAL ID AND RECOGNITION REQUEST 
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IMAGE PROCESSING SERVER 15 SWITCHES 
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ADAPTER CARD 39 TO CAMERA 1 8 AND ACTIVATES CAMERA 1 8 
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-182 



FEATURE EXTRACTOR 48 EXTRACTS 
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FOR ITEM 22 FROM FEATURE DATABASE 33 
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TRANSACTION SERVER 16 ADDRESSED TO 
TERMINAL 14 AND CONTAINING THE SKU NUMBER 
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TERMINAL 14 ADDS THE DESCRIPTION AND PRICE TO THE TRANSACTION I 

( END ) 200 
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